PhD

related works.tex (4907B)

---

 \section{Related Work}
 \label{sec:fitb:related work}
 The \textsc{nlp} and knowledge base related work is presented in Chapter~\ref{chap:context}, and the relation extraction related work is presented in Chapter~\ref{chap:relation extraction}.
 The main approaches we built upon are:
 \begin{itemize}
 	\item Distant supervision (Section~\ref{sec:relation extraction:distant supervision}, \cite{distant}): the method we use to obtain a supervised dataset for evaluation;%
 		\sidenote{As explained in Section~\ref{sec:relation extraction:clustering}, this is sadly standard in the evaluation of clustering approaches.}
 	\item \textsc{pcnn} (Section~\ref{sec:relation extraction:pcnn}, \cite{pcnn}): our relation classifier, which was the state-of-the-art supervised relation extraction method at the time;
 	\item Rel-\textsc{lda} (Section~\ref{sec:relation extraction:rellda}, \cite{rellda}): the state-of-the-art generative model we compare to;
 	\item \textsc{vae} for relation extraction (Section~\ref{sec:relation extraction:vae}, \cite{vae_re}): the overall inspiration for the architecture of our model, with which we share the entity predictor;
 	\item Self\textsc{ore} (Section~\ref{sec:relation extraction:selfore}, \cite{selfore}): an extension of our work, which, alongside their own approach, proposed an improvement of our relation classifier by replacing the \textsc{pcnn} by a \bertcoder{}.
 \end{itemize}
 In this section, we give further details about the relationship between our losses and the ones derived by \textcite{vae_re}.
 As a reminder, their model is a \textsc{vae} defined from an encoder \(Q(r\mid \vctr{e}, s; \vctr{\phi})\) and a decoder \(P(\vctr{e}\mid r, s; \vctr{\theta})\) as:
 \begin{marginparagraph}
 	The prior of a conditional \textsc{vae} \(P(r\mid\vctr{\theta})\) is usually conditioned on \(s\) too.
 	However, this additional variable is not used by \textcite{vae_re}.
 \end{marginparagraph}
 \begin{equation}
 	\loss{vae}(\vctr{\theta}, \vctr{\phi}) = \expectation_{Q(r\mid \vctr{e}, s; \vctr{\phi})}[ - \log P(\vctr{e}\mid r, s; \vctr{\theta})] + \beta \kl(Q(r\mid \vctr{e}, s; \vctr{\phi}) \mathrel{\|} P(r\mid\vctr{\theta}))
 	\label{eq:fitb:vae full loss}
 \end{equation}
 This is simply a rewriting of the \textsc{elbo} of Equation~\ref{eq:relation extraction:elbo} substituting relation extraction variables to the generic ones.
 There is however two differences compared to a standard \textsc{vae}.
 First, the variable \(s\) is not reconstructed, it simply conditions the whole process.
 Second, the regularization term is weighted by a hyperparameter \(\beta\).
 This makes the model of \textcite{vae_re} a conditional \(\beta\)\textsc{-vae} \parencitex{conditional_vae, beta_vae}[-11mm].
 The first summand of Equation~\ref{eq:fitb:vae full loss} is called the reconstruction loss since it reconstructs the input variable \(\vctr{e}\) from the latent variable \(r\) and the conditional variable \(s\).
 Since we followed the same structure for our model, this reconstruction loss is actually \loss{ep}, the difference being in the relation classifier.
 We can then rewrite the loss of \textcite{vae_re} as:
 \begin{marginparagraph}
 	As explained Section~\ref{sec:relation extraction:vae}, \(Q\) is the \textsc{vae}'s encoder.
 \end{marginparagraph}
 \begin{align*}
 	\loss{vae}(\vctr{\theta}, \vctr{\phi}) & = \loss{ep}(\vctr{\theta}, \vctr{\phi}) + \beta \loss{vae reg}(\vctr{\theta}, \vctr{\phi}) \\
 	\loss{vae reg}(\vctr{\theta}, \vctr{\phi}) & = \kl(Q(\rndm{r}\mid \rndmvctr{e}; \vctr{\phi}) \mathrel{\|} P(\rndm{r}\mid\vctr{\theta}))
 \end{align*}
 In their work, they select the prior as a uniform distribution over all relations \(P(\rndm{r}\mid\vctr{\theta}) = \uniformDistribution(\relationSet)\) and approximate \loss{vae reg} as follow:
 \begin{equation*}
 	\loss{vae reg}(\vctr{\phi}) = \expectation_{(\rndm{s}, \rndmvctr{e})\sim \uniformDistribution(\dataSet)} \left[ - \entropy(\rndm{R} \mid \rndm{s}, \rndmvctr{e}; \vctr{\phi}) \right]
 \end{equation*}
 Its purpose is to prevent the classifier from always predicting the same relation, i.e.~it has the same purpose as our distance loss \loss{d}.
 However, its expression is equivalent to \(-\loss{s}\), and indeed, minimizing the opposite of our skewness loss increases the entropy of the classifier output, addressing \problem{2} (classifier always outputting the same relation).
 Yet, using \(\loss{vae reg}=-\loss{s}\) alone, draws the classifier into the other pitfall \problem{1} (not predicting any relation confidently).
 In a traditional \textsc{vae}, \problem{1} is addressed by the reconstruction loss \loss{ep}.
 However, at the beginning of training, the supervision signal is so weak that we cannot rely on \loss{ep} for our task.
 The \(\beta\) weighting can be decreased to avoid \problem{1}, but this would also lessen the solution to \problem{2}.
 This causes a drop in performance, as we show experimentally.